feat: support whole-value Variant scan pushdown - #6443

Summary

• Prior state and problem: Spark’s whole-value Variant scan rewrite caused native scans to fall back, including under Spark 4.1’s default configuration.
• Design approach: Recognize exactly one RequestedVariantField.fullVariant, carry its identity through protobuf and Arrow metadata, and reuse the existing Variant normalizer.
• Correctness / compatibility analysis: Request metadata matches Spark sources across 4.0.0, 4.0.4, 4.1.0, 4.1.3 and 4.2.0. Null handling and Spark’s wrapper layout agree. Typed, nested and multi-field requests retain fallback. No introduced P1/P2 issues found within this review.
• Key design decisions: The dedicated request flag keeps serialization narrow. The wrapper shares normalized buffers and validity data without another full-column copy. Existing FFI ownership and row-conversion paths remain applicable.
• Implementation sketch: Scan admission validates the request, serde preserves its identity, the schema adapter normalizes and wraps the column, and JVM import restores metadata for Spark’s projection.
• Behavioral changes worth calling out: Compared with branch-1.1, whole-value rewritten scans intentionally become native, and shredded objects containing empty keys gain a reconstruction fix. The existing normalizer’s cost now also applies to rewritten scans. No performance benchmark was run.
• Suggested improvements: No P1/P2 code changes to request.

Reviewed all 18 changed files at e02715d6d868f4da4707f077bc048e2ad5dbbf66 against base fef94f6cd78b18151dff57b7a936798385356de5. GitHub confirms non-draft status. The snapshot and live discussion endpoints contain no existing review concerns. Routed skills: review-comet-pr, review-comet-expression-pr, and review-comet-ffi-pr.

Exact-head CI: run 37207859295 passed native tests, including all three new Rust regression tests, and Spark 4.1 scans, including the new whole-value and fallback assertions. Shuffle, TPC-H, native build and profile compilation checks also passed. Expressions, execution and TPC-DS remain running. No failed checks were reported.

Validation limits: Spark SQL jobs were skipped, and nondefault Spark profile runtime tests were not run. The local native attempt failed because HDFS compilation could not find jni.h. A subsequent build without HDFS reached its 180-second limit before executing tests. No local JVM tests completed. Broader compatibility validation remains outstanding.